The following table summarizes functionalities of replay buffers.
cpprb | OpenAI/Baselines | Ray/RLlib | Chainer/ChainerRL | DeepMind/Reverb | |
---|---|---|---|---|---|
Flexible Environment | Yes | No | No | Yes | Yes |
Nstep | Yes | No | Yes | Yes | No |
Parellel Exploration | Yes (Support Ape-X on single machine) | Yes (Avarages gradients of MPI processes) | Yes (Concatenates sample batches from distributed buffers) | No | Yes |
Save/Load | Only transitions (document) | No (Maybe can pickle ) |
No (Maybe can pickle . Trained policies can save/load.) |
Yes | Yes |
Deep Learning Framework | Anything | TensorFlow 1.14 (only this version) | Anything (Helper functions for TensorFlow and PyTorch) | Chainer (maintenance only) | TensorFlow 2.3 |